The present application relates to an image-data processing apparatus suitable for image processing carried out on a picked-up image by using a digital video camera or the like and relates to an image-data processing method adopted in the image-data processing apparatus.
An image pickup apparatus such as a video camera employs an image memory used for temporarily storing picked-up image data. By making use of such an image memory, many image pickup apparatus are capable of carrying out image-processing functions such as a function to convert data stored in the image memory as the data of a picked-up image into output image data having a desired resolution, a function to compensate the picked-up image data for an electronic effect caused by a hand movement and an function to execute effect processing such as the so-called electronic zoom.
As a traditional method for processing image data by making use an image memory of this type, an image-data processing method is described in U.S. Pat. No. 2,956,527 used as patent document 1. In accordance with the image-data processing method described in patent document 1, image processing is carried out by splitting the processing into a horizontal-direction image process and a vertical-direction image process. The image-data processing method disclosed in patent document 1 is explained by referring to FIG. 54 as follows.
As shown in FIG. 54, in accordance with write control based on a write/read control signal W/R generated by a control unit 301, data input from an imager as the data of picked-up image is stored in an image memory 302 having a storage capacity large enough for holding picked-up image data with a quantity equivalent to one screen. The picked-up image data is a digital image signal obtained as a result of an A/D process to convert an original analog signal. In the following description, a hardware unit such as the control unit 301 is also referred to as a section.
After the lapse of a vertical-direction synchronization period, the data stored in the image memory 302 as the picked-up image data is read out from the memory unit 301 in accordance with read control based on a write/read control signal W/R generated by the control unit 301 and supplied to a horizontal-direction image processing unit 303 for carrying out horizontal-direction image processing such as a horizontal-direction resolution conversion process and a process to compensate the data for an effect caused by a horizontal-direction hand movement. To put it in detail, the horizontal-direction image processing unit 303 is a section for carrying out pixel skipping and pixel interpolation processes on the picked-up image data in order to make the resolution in the horizontal direction equal to a specified value or compensate the data for an effect caused by a hand movement occurring in the horizontal direction.
The picked-up image data completing the horizontal-direction image processing carried out by the horizontal-direction image processing unit 303 is supplied to a vertical-direction image processing unit 304. By the same token, the vertical-direction image processing unit 304 is a section for carrying out vertical-direction image processing such as a vertical-direction resolution conversion process and a process to compensate the data for an effect caused by a vertical-direction hand movement. To put it in detail, the vertical-direction image processing unit 304 is a section for carrying out pixel skipping and pixel interpolation processes on the data of the picked-up image in order to make the resolution in the vertical direction equal to a specified value or compensate the data for an effect caused by a hand movement occurring in the vertical direction. Thereafter, a signal of an output image in a desired resolution performed such as a compensation of hand movement is output.
As described above, the image memory 302 has a storage capacity large enough for holding picked-up image data with a quantity at least equivalent to one screen or equivalent to one vertical-direction period. As shown by a solid inclined line in FIG. 55, the data of a picked-up image is stored in the image memory 302. As shown by a dashed inclined line in FIG. 55, on the other hand, the data of a picked-up image is read out from the image memory 302 with a timing delayed from the operation to store the data in the image memory 302 by one vertical-direction period. As shown in the figure, the operation to store the data of a picked-up image in the image memory 302 and the operation to read out the data from the image memory 302 are carried out repeatedly.
The image memory 302 has a storage capacity large enough for holding picked-up image data with an quantity at least equivalent to one screen because, in this typical configuration, the data of a picked-up image is read out from the image memory 302 with a timing delayed from the operation to store the data in the image memory 302 by one vertical-direction period. That is to say, after the lapse of one vertical-direction synchronization period, picked-up image data stored in the image memory 302 would be undesirably overwritten by picked-up image data of the next vertical-direction synchronization period. Thus, in order to avoid the overwriting of the imager-created image data stored in the image memory 302 before the data is read out from the image memory 302, the data must be read out from the image memory 302 before the operation to store the data of the next vertical-direction synchronization period is started. That is to say, the image memory 302 is required to have a margin in the storage capacity as a margin added to the capacity equivalent to one screen if the data is not read out from the image memory 302 before the operation to store the data of the next vertical-direction synchronization period is started.
By providing the image memory 302 with such a storage capacity, data of a picked-up image can be read out from the image memory 302 while preventing a read operation from surpassing an outgoing write operation. Thus, the process to convert the resolution of the image can be carried out to generate image data having attributes including a desired resolution.
In accordance with the image-data processing method described in patent document 1, however, the image memory 302 is required to have a storage capacity corresponding to a highest-resolution pixel count of the imager. The pixel count is defined as a product obtained by multiplying the number of pixels per horizontal line by the number of horizontal lines. As an example, let us assume data of an HD (High Definition) image. In this case, if luminance and color-difference signals of one field of the HD image data are stored in a memory as eight bits per pixel, the image memory 302 is required to have a storage capacity of about 12M bits. Thus, the image memory 302 has a large size and is also expensive.
In addition, an imager having a larger pixel count raises a problem of a required large storage capacity of the image memory 302.
As an alternative configuration, the horizontal-direction image processing unit 303 is provided at a stage in front of the image memory 302 as shown in FIG. 56. In this alternative configuration, the picked-up image data from the imager is not stored directly in the image memory 302 as it is. Instead, in accordance with a method conceivable for this alternative configuration, the data of the picked-up image is subjected to horizontal-direction image processing such as a horizontal-direction resolution conversion process in the horizontal-direction image processing unit 303 in order to reduce the horizontal-direction amount of the data before the data of one screen is stored in the image memory 302. Also in the case of the method provided for this alternative configuration, the data of a picked-up image is read out from the image memory 302 with a timing delayed from the operation to store the data in the image memory 302 by one vertical-direction period. Then, the picked-up image data read out from the image memory 302 is supplied to the vertical-direction image processing unit 304 for carrying out vertical-direction image processing such as a vertical-direction resolution conversion process and a process to compensate the data for an effect caused by a vertical-direction hand movement.
In the case of the typical configuration shown in FIG. 56, the image memory 302 is used for storing one-screen picked-up image data having a pixel count reduced by the horizontal-direction image processing carried out by the horizontal-direction image processing unit 303. For this reason, the image memory 302 is required to have a storage capacity for accommodating data with an amount merely equivalent to one screen after the horizontal-direction image processing. Thus, the storage capacity of the image memory 302 employed in the configuration shown in FIG. 56 can be reduced to a small value in comparison with the image memory 302 employed in the configuration shown in FIG. 54.
By the way, in an operation to store data completing the horizontal-direction image processing in the horizontal-direction image processing unit 303 and an operation to read out the data from the image memory 302 to be subjected to the vertical-direction image processing in the configuration shown in FIG. 56, the following problem may be raised, depending on the substances of the horizontal-direction image processing and the vertical-direction image processing.
In the vertical-direction image processing carried out in the configuration shown in FIG. 56, a request REQ indicating a horizontal line whose data is about to be processed is supplied by the vertical-direction image processing unit 304 to the control unit 301. On the basis of the horizontal line indicated by the request REQ, the control unit 301 determines a plurality of horizontal lines to be used in processing such as a vertical-direction interpolation process for computing the picked-up image data of the horizontal line indicated by the request REQ and identifies the addresses of the horizontal lines in the image memory 302. Then, the control unit 301 reads out the picked-up image data of the determined horizontal lines from the identified addresses in the image memory 302 as data required in the vertical-direction image processing such as a vertical-direction interpolation process for computing the picked-up image data of the horizontal line indicated by the request REQ. In the following description, the horizontal line indicated by the request REQ is referred to as a requested horizontal line.
In this case, for example, when a horizontal line whose data is about to be processed by the vertical-direction image processing unit 304 is determined, that is, when the requested horizontal line is determined, the addresses (strictly speaking, vertical-direction addresses) of the horizontal lines to be used in vertical-direction processing such as a vertical-direction interpolation process for computing the picked-up image data of the determined horizontal line do not generally change in the horizontal-direction in the image processing such as an electronic hand movement distortion compensation process, an electronic zoom process and a resolution conversion process as shown in FIG. 57.
Thus, when the request REQ indicating a horizontal line whose data is about to be processed arrives at the control unit 301, in processing of picked-up image data of one horizontal line, by specifying the same vertical-direction address of any individual one of the horizontal lines to be used in vertical-direction image processing, the control unit 301 is capable of reading out the data of the picked-up image for the individual horizontal line from the image memory 302.
When the request REQ indicating a horizontal line whose data is about to be processed arrives at the control unit 301 in the course of image processing, however, the control unit 301 does not merely specify the same vertical-direction address in an operation to read out picked-up image data of one horizontal line from the image memory 302, but must also change the vertical-direction address in accordance with the horizontal-direction position of the pixel on the horizontal line. An example of this case is explained as follows.
In general, an image pickup lens has a distortion characteristic, which distorts a picked-up image as typically shown in FIG. 58. If a high-precision and high-performance image pickup lens is employed, the distortion is not noticeable. If a cheap image pickup lens or an optical zoom lens is used, however, unignorable distortion is generated in the image.
For example, if an image of a lattice stripe pattern like one shown in FIG. 58A as a stripe pattern serving as an object of photographing is taken in a photographing operation by using an image pickup lens having a much-distortion characteristic, the horizontal-direction and vertical-direction straight horizontal lines composing the lattice stripe pattern of the image are bent due to distortion generated by the much-distortion characteristic in the image as shown in FIG. 58B. The image distortion caused by the image pickup lens depends on the position of every pixel in the imager and is peculiar to the image pickup lens, varying from lens to lens. In addition, the image distortion caused by the image pickup lens also depends on the position of the lens. That is to say, when the position of the image pickup lens is changed, the image distortion caused by the image pickup lens also varies in accordance with the change in position.
Also if a process to compensate an image for distortion caused by an image pickup lens as described above is carried out, it is possible to think of a method by which the distortion is split into horizontal-direction image distortion and vertical-direction image distortion. Then, the horizontal-direction image processing unit 303 compensates the image for the horizontal-direction image distortion and stores the result of the compensation process in the image memory 302. Subsequently, on the basis of a horizontal line request REQ received from the vertical-direction image processing unit 304, the control unit 301 determines a plurality of horizontal lines to be used in vertical-direction image processing and reads out data of a picked-up image from the image memory 302.
However, for a requested horizontal line in this case, it is necessary to carry out a process to change the vertical-direction address of a pixel on any one of the horizontal lines to be used in vertical-direction image processing in accordance with the horizontal-direction position of the pixel as shown in FIG. 59. That is to say, after the vertical-direction image processing, the pixel data of the requested horizontal line is shifted to a correct location in the vertical direction from an incorrect position in the image prior to the vertical-direction image processing as shown in the left-side portion of FIG. 59.
The vertical-direction image processing is an interpolation process carried out by making use of a plurality of horizontal lines above and beneath the requested horizontal line. In this case, the vertical-direction address of any specific pixel included in each of a plurality of horizontal lines above and beneath pixels on the requested horizontal line on an image prior to the vertical-direction image processing changes in accordance with the horizontal-direction position of a pixel located on the requested horizontal line as a pixel corresponding to the specific pixel. That is to say, the vertical-direction address associated with a storage location in the image memory 302 as an address required in the vertical-direction image processing to be carried out on a horizontal line is not an address fixed for the processing to be carried out on the horizontal line, but an address changing randomly in accordance with the horizontal-direction position of a pixel located on the requested horizontal line.
As a result, in the vertical-direction image processing carried out as a part of processing to compensate a picked-up image for distortion caused by the image pickup lens, data of the image is read out from a horizontal-line position located in the image memory 302 as a position varying randomly in accordance with the horizontal-direction position of a pixel located on the requested horizontal line. That is to say, random accesses are required.
FIG. 60 is an explanatory diagram to be referred to in description of an easiest method for avoiding random accesses in an operation to read out the data of a picked-up image from the image memory 302. In accordance with this method, first of all, attention is paid to the requested horizontal line. Then, the picked-up image is searched for the uppermost and lowermost ones of a plurality of horizontal lines shown in the figure as horizontal lines required in the vertical-direction image processing to be carried out on the requested horizontal line. Finally, the data of all the horizontal lines in a range between the uppermost and lowermost ones is read out from the image memory 302.
In accordance with this method, while the addresses of the horizontal lines whose data is to be read out from the image memory 302 can be computed with ease, data of a large number of pixels not required in the vertical-direction image processing (in FIG. 60, unpainted spaces surrounded by dashed grid lines) is inadvertently read out from the image memory 302. Thus, the power consumption is increased by a quantity corresponding to the operation to inadvertently read out the data of a large number of pixels not required in the vertical-direction image processing from the image memory 302. If the worst comes to worst, in order to carry out real-time processing, the clock frequency must be increased.
As shown in FIG. 61, on the other hand, for every pixel on the requested horizontal line, a plurality of horizontal-line addresses required in the vertical-direction image processing are computed and pixel data required in the vertical-direction image processing is read out from the horizontal-line addresses required in the vertical-direction image processing. In this way, data of pixels not required in the vertical-direction image processing is never read out from the image memory 302. Thus, the problem of the method explained above by referring to FIG. 60 is not raised.
However, in the case of the method explained above by referring to FIG. 61, it is necessary to compute a plurality of horizontal-line addresses required in the vertical-direction image processing for every pixel on the requested horizontal line and, in addition, issue a read command for each of the horizontal-line addresses. Thus, from the power-consumption point of view, the method explained above by referring to FIG. 61 is also impractical.
Addressing the problems described above, inventors presented an image-data processing method and an image-data processing apparatus, which are capable of solving the power-consumption and processing-efficiency problems described above as described in Japanese Patent Laid-open No. 2006-222827 used as patent document 2.
In accordance with the image-data processing method disclosed in patent document 2, a picked-up image is not searched for the uppermost and lowermost ones of a plurality of whole horizontal lines associated with the requested horizontal line as horizontal lines required in the vertical-direction image processing as is the case with the method explained earlier by referring to FIG. 60. Instead, the image is searched for the uppermost and lowermost ones of a plurality of partial horizontal lines associated with the requested horizontal line as horizontal lines required in the vertical-direction image processing. In this case, a partial horizontal line is a horizontal line portion composed of a plurality of pixels arranged in the horizontal-direction. The number of pixels arranged in the horizontal-direction is equal to 1/N of effective pixels on the entire horizontal line where N is a positive integer at least equal to 2. The process to search the image for the uppermost and lowermost ones of a plurality of partial horizontal lines each composed of a plurality of pixels arranged in the horizontal-direction is carried out for each plurality of such partial horizontal lines. That is to say, the process to search the image for the uppermost and lowermost ones of a plurality of partial horizontal lines each composed of a plurality of pixels arranged in the horizontal-direction is carried out N times for the whole horizontal line.
FIG. 62 is a diagram showing a typical case in which the process to search the image for the uppermost and lowermost ones of a plurality of partial horizontal lines each composed of a plurality of pixels arranged in the horizontal direction is carried out for each partial horizontal line composed of eight pixels arranged in the horizontal direction.
In accordance with the method disclosed in patent document 2, data of a picked-up image is stored in an area of an image memory or read out from the area in horizontal line segments each including a plurality of pixels. Thus, in comparison with a method for reading out image data of a plurality of whole horizontal lines as is the case with the method explained earlier by referring to FIG. 60, it is possible to reduce the number of pixels whose data is read out from an area of an image memory as data not required in the vertical-direction image processing. As a result, the processing efficiency is improved.
To put it in detail, after vertical-direction distortion variations of the eight pixels arranged in the horizontal-direction are checked, the access range is determined in order to limit vertical-direction accesses to the image memory to accesses to only a minimum required range in the eight pixels arranged in the horizontal direction. It is thus possible to decrease the number of operations to read out data not required in the vertical-direction image processing from the image memory. As is evident from FIG. 62, it is possible to decrease the number of operations to read out data not required in the vertical-direction image processing from the image memory in comparison with a method for reading out image data of a plurality of whole horizontal lines as is the case with the method explained earlier by referring to FIG. 60.
In addition, in accordance with the method disclosed in patent document 2, data of a picked-up image is read out from the image memory in horizontal line segments each including a plurality of pixels, for example, in horizontal line segments each including eight pixels. Thus, a read command is issued for every plurality of pixels. As a result, in comparison with a method whereby a read command is issued for each pixel as is the case with the method explained earlier by referring to FIG. 61, the number of operations to issue a command can be reduced, allowing the power consumption to be also decreased as well.
As is evident from what is described above, the method disclosed in patent document 2 is capable of solving the problems raised by the methods explained by referring to FIGS. 60 and 61. It is thus possible to implement reduction of the power consumption and improvement of the processing efficiency.
By the way, the image memory 302 employed in the processing configuration shown in FIG. 56 can be divided into a first memory area 3021 used for storing a processing result output by the horizontal-direction processing unit 303 and a second memory area 3022. FIG. 63 is a block diagram showing a configuration, which includes the horizontal-direction image processing unit 303, the first image memory area 3021, the vertical-direction image processing unit 304 and the second image memory area 3022, as well as detailed write and read accesses made to the first image memory area 3021 and the second image memory area 3022.
As shown in FIG. 63, an access W1 is a process to store image data obtained as a result of horizontal-direction image processing carried out by the horizontal-direction image processing unit 303 into the first image memory area 3021 of the image memory 302. An access R1 is a process to read out the image data obtained as a result of the horizontal-direction image processing carried out by the horizontal-direction image processing unit 303 from the first image memory area 3021 of the image memory 302. An access W2 is a process to store image data obtained as a result of vertical-direction image processing carried out by the vertical-direction image processing unit 304 into the second image memory area 3022 of the image memory 302. An access R2 is a process to read out the image data obtained as a result of the vertical-direction image processing carried out by the vertical-direction image processing unit 304 from the second image memory area 3022 of the image memory 302.
The horizontal-direction and vertical-direction image processing of input data of a picked-up image as well as processing to produce output data of the image may need to be carried out in a real-time manner as is the case with processing to record and/or reproduce data of a moving image for example. In the case of such real-time manner processing, the four accesses W1, W2, R1 and R2 need to be made to every horizontal-direction multi-pixel line segment consisting of eight pixels in accordance with the image-data processing method adopted in the image-data processing apparatus disclosed in patent document 2.
In accordance with the image-data processing method adopted in the image-data processing apparatus disclosed in patent document 2, an access to one pixel is made in one memory-access cycle, which is referred to hereafter as a cycle. Thus, an access to a write/read unit, which is the aforementioned horizontal line segment consisting of eight pixels, is made in eight cycles referred to as one processing time unit period. Each cycle of one processing time unit period is allocated to one of the four accesses W1, W2, R1 and R2 for example as shown in FIG. 64. That is to say, each individual one of the four accesses W1, W2, R1 and R2 is made in a cycle allocated to the individual access.
One of the four accesses W1, W2, R1 and R2 is the read access R1. As described before, the read access R1 is made in order to read out not only eight pixels arranged in the horizontal-direction, but also eight pixels arranged in the vertical-direction for each of the eight pixels arranged in the horizontal-direction. Thus, one cycle allocated to the read access R1 is not sufficient. By allocating one cycle to each of the three remaining accesses W1, W2 and R2, however, each of the accesses can be made.
In order to solve the problem described above, in accordance with the image-data processing method adopted in the image-data processing apparatus disclosed in patent document 2, 3 of the eight cycles in one processing time unit period are allocated to the three accesses W1, W2 and R2 respectively whereas up to five cycles are allocated to the read access R1 as shown in FIG. 64.
In addition, the image memory 302 is configured to include four banks and the control unit 301 is capable of carrying out an operation to read out data of a picked-up image from each of the banks of the image memory 302 in horizontal line units as well as an operation to store such image data into each of the banks of the image memory 302 also in horizontal line units. On top of that, in a read access R1, four horizontal lines can be read out from all the four banks of the image memory 302 at the same time.
As is evident from the above description, up to five access rights are given to the read access R1 made by the control unit 301 to read out image data required in the vertical-direction image processing in one processing time unit period, allowing four horizontal lines to be read out from all the four banks of the image memory 302 at the same time for each of the given access rights. Thus, image data of 20 (=4 horizontal lines/access right×5 access rights) horizontal lines can be acquired during one processing time unit period.
Accordingly, in one processing time unit period, vertical-direction image distortion (or a vertical-direction variation) can be dealt with in real-time processing in horizontal-direction eight-pixel units as shown in FIG. 65 provided that the distortion is in a distortion range of up to a variation 12 (which is a vertical-direction variation of 12 pixels). As described earlier, one processing time unit period consists of eight consecutive cycles, which form a range executable in a real-time manner.
That is to say, in an image pickup apparatus according to this embodiment, if the vertical-direction variations in the eight pixels arranged in the horizontal-direction are variations having an amplitude of 12 pixels, processing to compensate the image for the variation can be carried out in a real-time manner. It is to be noted that slots 1 to 5 in the example shown in FIG. 65 each correspond to one access. In the following description, the amplitude of 12 pixels is referred to as a variation amplitude of 12.
As described above, if the lens distortion has a magnitude not exceeding a value determined in advance, that is, if the lens distortion is variations having an amplitude smaller than the variation amplitude of 12 in the case of the example described above, it is possible to carry out the processing to compensate an image for the distortion in order to result in an image having a high quality and implement a resolution conversion process. In the case of a lens employed in an ordinary digital camera or an ordinary video camera, the variations caused by image distortion are variations having an amplitude smaller than the variation amplitude of 12. Thus, such a variation can be dealt with sufficiently.
However, let us assume for example that an image pickup apparatus is provided on the rear part of a car and an image taken by the image pickup apparatus is referred to in an operation to drive back the car in order to park the car in a garage. In this case, a super-wide angle lens such as a fish-eye lens is employed in the image pickup apparatus. If a special lens such as a fish-eye lens is employed in the image pickup apparatus, variations caused by image distortion may be variations having an amplitude greater than the variation amplitude of 12, raising a problem that it is impossible to read out required image data of a plurality of horizontal lines in a unit consisting of eight pixels arranged in the horizontal direction in one processing time unit period.